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Executive Summary 


C ompetency-based education (CBE) has garnered 
significant attention lately from reformers and 
policymakers. Put simply, CBE awards credit based on 
what students have learned rather than how much time 
they spend in class. Competency-based programs iden- 
tify specific competencies, develop assessments to mea- 
sure mastery of those competencies, and then award 
credit or other credentials to those students who meet 
or exceed benchmarks on those assessments. 

CBE programs give students flexibility to move at 
their own pace, which could potentially shorten the 
time to degree and enhance affordability. CBE creden- 
tials may also more clearly signal to employers students’ 
knowledge and career preparedness. 

Clearly, the benefits of expanding access to CBE 
could be substantial. But, what does existing research 
suggest about the likely effect of reforms to promote 
CBE? In this paper, we analyze 380 studies of postsec- 
ondary CBE and prior-learning assessment listed in 
the Department of Educations Education Resources 
Information Center database. We reviewed each study’s 
methodology (i.e., quantitative or qualitative) and 
topic (i.e., program design, student characteristics, stu- 
dent outcomes, and policy environment). 

We found that existing research leaves important 
questions unanswered. What types of students enroll in 
CBE? Blow do students fare in CBE programs, and do 
particular groups do better than others? Are CBE grad- 
uates more attractive to potential employers? 

Our analysis uncovered more than twice as many 
qualitative studies (228 articles) as quantitative ones 
(102). The studies in our sample tended to focus on 
questions of design and practice, describing the man- 
ner in which providers have identified competencies, 
developed assessments, and structured courses and 
programs. Far fewer articles explored data on ques- 
tions about who actually enrolls in CBE and how 


students fare in terms of learning, completion, and 
labor market success. A substantial number of stud- 
ies (56) examined assessment in CBE, including ques- 
tions about the design, results, and validity of different 
assessment techniques. 

We also looked at the relationship between method- 
ology and content. We found that qualitative studies 
typically described program design (113) or prescribed 
best practices for program design (97). Quantitative 
studies examined program design (47) and student out- 
comes (42). 

However, the research on outcomes was limited. 
Many of these studies looked at students’ self-assessments 
of their own competencies after a competency-based 
course or program. Few examined outcomes such as 
retention, graduation, or job-placement rates. Only a 
handful looked at CBE in comparison to a counterfac- 
tual; just 13 compared CBE outcomes to those from 
traditional programs. 

The paper concludes with some recommenda- 
tions for future research. We suggest that researchers 
should use the ongoing expansion of CBE programs 
as an opportunity to launch a research and develop- 
ment agenda. Specifically, we suggest that researchers 
work with CBE providers to answer the following basic 
questions: 

• How do the demographics of students who enroll 
in CBE compare to those enrolled in traditional 
programs? 

• What do success rates in CBE programs look 
like, especially relative to comparable programs? 
Do students who earn credit via prior-learning 
assessment perform comparably in subsequent 
coursework? 
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• How do employers view CBE graduates? Do they 
see CBE credentials as being more informative 
than traditional degrees? 


We sketch out the role that policymakers, philan- 
thropists, and other parties should play in facilitating 
the rigorous evaluation of CBE programs. 
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This paper is the sixth in a series examining competency-based higher education from a number of perspectives. 


H igher education is under increasing pressure to 
change. The combination of soaring costs, stag- 
nant completion rates, and an uncertain labor market 
for recent college graduates has given rise to a sustained 
push for innovation, both at existing colleges and via 
new postsecondary providers. 

In the ensuing national debate, few ideas have 
received more attention than competency-based edu- 
cation (CBE), which has been around for decades but 
has reemerged as a favorite innovation of reformers 
and policymakers. Simply put, competency-based 
models award credit based on what students can prove 
they have learned rather than how much time they 
spend in class. CBE programs identify specific compe- 
tencies, develop assessments to measure student mas- 
tery of those competencies, and then award academic 
credit to students who meet or exceed benchmarks on 
those assessments. 

The model’s appeal derives largely from its poten- 
tial to overcome some of the pressing challenges fac- 
ing higher education. First, it could enhance college 
affordability by allowing students to progress through a 
degree program at their own pace, potentially decreas- 
ing the time it takes some students to earn a credential. 1 
Second, CBE could help boost college completion, as 
it enables prospective students to obtain credit for prior 
learning — skills and knowledge learned on the job, in 
the military, and so on — thereby reducing the number 
of credits needed to graduate. 2 Third, a credential from 
a CBE program may better signal what a student knows 
and is able to do than a traditional college transcript. 


Because competencies are clearly identified and credit is 
awarded based on mastery, employers could more accu- 
rately distinguish the students who have the skills and 
knowledge they need. 3 Lastly, a focus on transparent 
measures of student learning instead of time could help 
move us toward a market in which schools compete on 
the quality of the education they provide rather than 
the students they enroll or the amenities they offer. 4 

The potential to improve these dimensions is rea- 
son enough to experiment with CBE models. Luck- 
ily, a growing number of institutions are doing so 
already. And policymakers at the federal and state 
level have worked to create space for such models, 
inviting providers into state systems and waiving 
existing regulations for a subset of programs so as 
to study their outcomes. Indiana, for instance, wel- 
comed CBE provider Western Governors University 
(WGU) as an official state institution in 2010, which 
allowed Indiana students to enroll using state finan- 
cial aid. 5 Other states have incorporated WGU into 
their systems too, including Missouri, Nevada, Ten- 
nessee, Texas, and Washington. 6 At the federal level, 
a 2014 bill passed unanimously in the Elouse of Rep- 
resentatives setting up “demonstration projects” for 
several CBE providers, waiving federal aid require- 
ments such as the credit hour. 7 

What does existing research suggest about the likely 
effect of expanding CBE on access, affordability, and 
attainment? Who enrolls in CBE programs? Who com- 
pletes them, how do they fare after graduation, and how 
do their outcomes compare to similarly qualified peers? 
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To shed some light on the stock of CBE research — 
with an eye toward nudging researchers and institutions 
to adopt a proactive research agenda — we analyzed 
studies listed in the Department of Educations Edu- 
cation Resources Information Center (ERIC) database. 
We limited our analysis to those that focused on CBE 
at the postsecondary level and were published between 
1996 and 2015. 

We find that existing research leaves many of these 
basic questions unanswered. In short, researchers have 
not conducted the kinds of rigorous empirical evalua- 
tions of CBE that, in a perfect world, leaders and pol- 
icymakers could draw on in creating effective policies. 
The majority of CBE studies in the ERIC database 
employ a qualitative methodology, while quantitative 
studies are less common. Only a small portion of the 
studies reviewed relevant literature. There are more 
than two times as many qualitative articles as quanti- 
tative articles. 

Most studies in the database focus on questions of 
program design and institutional practice, describing 
the process through which institutions have identi- 
fied competencies, created assessments, and developed 
programs before making recommendations about best 
practices. Fewer articles have explored empirical ques- 
tions as to who actually enrolls in CBE and what pro- 
gram outcomes such as learning, completion, and labor 
market success look like. 

Some of this gap reflects the newness of the latest 
round of experimentation; dozens of programs have 
only recently emerged, and research on them is just get- 
ting underway. This flurry of experimentation represents 
a key opportunity to engage in the kind of research and 
development necessary to inform policy and institu- 
tional decision making. We conclude the paper with 
some recommendations for such an agenda, as well as 
advice to different stakeholders who can help develop it. 

The rest of this paper proceeds as follows. First, we 
describe the process by which we identified and coded 
studies in the ERIC database. We then explore the find- 
ings, looking first at methodology (quantitative ver- 
sus qualitative) and then at content (what the studies 
focused on). We conclude with some suggestions for 
future research and the role that policymakers and other 
parties might play in facilitating that research agenda. 


Methodology: Identifying and Coding Studies 

To get a sense of the existing research on postsecond- 
ary CBE, we turned to the ERIC database, an online 
bibliographic and full-text digital library of education 
research, which includes peer-reviewed journal arti- 
cles and non-journal materials. 8 The Institute of Edu- 
cation Sciences administers the database. ERIC selects 
its peer-reviewed journal articles from more than 1,000 
academic journals from around the world, and its col- 
lection includes bibliographic records for 1.6 million 
items indexed since 1966. 9 It is the largest and most 
widely used database for education-related literature. 10 

To identify studies of CBE in a postsecondary con- 
text within the ERIC database, we searched for stud- 
ies using the term “competency based education.” 11 
We also searched using the term “prior learning assess- 
ment” — a type of CBE — which generated an addi- 
tional set of studies (44 articles) that did not surface in 
our search of “competency based education.” 12 

We further refined the results in a few ways. First, 
we considered papers from only the last 20 years. Sec- 
ond, we included only peer-reviewed articles. Some may 
quibble with that choice, but peer-reviewed publications 
are considered the gold standard in scientific research. 

We excluded papers that exclusively analyzed CBE 
in an elementary or secondary school setting. We 
also excluded professional development and on-the- 
job training that was not done in conjunction with a 
college or university. 13 For instance, an article titled 
“Kentucky’s Early Childhood Professional Develop- 
ment Initiative to Promote Social Emotional Compe- 
tence” did not make the cut because it fell under the 
professional-development category. 14 Lastly, we elimi- 
nated clear false positives that had nothing to do with 
CBE. 1 5 

This search and the refinements yielded 380 aca- 
demic papers. While this sample likely does not cover 
all existing research on CBE — and obviously misses 
any non-peer-reviewed research — we believe it rep- 
resents the majority of recent, peer-reviewed research 
on the model. We encourage other researchers to use 
different search criteria, although we would be sur- 
prised if modified search techniques dramatically 
altered the overall findings. 


2 


INNOVATE AND EVALUATE 


ANDREW P. KELLY AND ROONEY COLUMBUS 


The ERIC search produced a list of article titles 
including information on authors, the year of publica- 
tion, the journal name and number, and the abstract. 
To review each article’s content, we first sought access 
to the full text. If the ERIC database had no link to 
the full text, we searched Google and Google Scholar 
for the title of the article in question. If that failed, we 
turned to the online research database EBSCOhost. 

These measures allowed us to locate the full text of 
80 percent (305 of 380) of the articles in our sample. 
For the papers we could not track down, we relied on 
the abstracts provided by ERIC, which are available 
for all articles. We then reviewed the full text or the 
abstracts to code studies according to their research 
methodology and their focus. 

We first coded the 380 articles according to the 
research methodology employed. Specifically, we inves- 
tigated whether the paper was primarily quantitative, 
qualitative, or a review of previous literature. A quan- 
titative paper relies primarily on data and statistics to 
analyze research questions. A qualitative paper uses 
interviews, descriptive case studies, and other nonsta- 
tistical techniques. A literature review presents an over- 
view of prior research in the field but does not offer 
new research findings of its own. 16 

We considered these three categories mutually exclu- 
sive, but we admit that there is potentially overlap. For 
instance, a researcher could employ mixed quantitative 
and qualitative methods when tackling a research ques- 
tion. Likewise, most empirical studies include a short 
literature review to situate their findings. To be sure we 
were correctly categorizing quantitative research, we 
erred on the side of coding mixed-method articles as 
quantitative. 17 

We also analyzed each study’s subject matter: what 
topics or research questions did the authors pursue? We 
categorized each paper as covering one or more of the 
following content categories. 

• Student Characteristics. Papers in this category 
describe the “who” of CBE: what types of stu- 
dents enroll in CBE programs? 

• Student Outcomes. Papers in this category 
describe the educational outcomes of CBE 


courses or programs. These papers could explore 
the outcomes — such as learning outcomes, reten- 
tion and graduation rates, or employment — of a 
specific CBE program in isolation or could com- 
pare the outcomes of CBE students to those of 
similar students not participating in CBE. 

• Assessment. Papers in this category focus on the 
process of developing and validating the assess- 
ments at the center of competency-based pro- 
grams. Some studies included in this category 
propose new tools for measuring competency, 
while others evaluate the validity of existing 
assessment methods — whether they actually mea- 
sure the competencies identified — and the align- 
ment of assessments to curriculum. 

• Description of Program Design. Papers in this 
category describe the details of a CBE program or 
programs, including program structure, mode of 
instruction, manner of assessment, and number 
of students. Oftentimes this type of study takes 
the form of a case study of a particular institution 
or program. 

• Prescription for Program Design. Papers in this 
category generally seek to outline best practices 
in designing CBE programs. This type of study 
recommends how to best construct, implement, 
or administer CBE. Several studies in this cate- 
gory focused on identifying competencies needed 
in a particular field, often through a survey of 
experts. 18 

• Policy and Market Environment. Papers in this 
category examine the CBE market and the pol- 
icy environment in which it exists. That includes 
articles on the number and type of CBE provid- 
ers, the laws and regulations that govern post- 
secondary CBE, and potential obstacles to CBE 
implementation. 

• Educational Theory. Papers in this category 
discuss the theory behind competency-based 
approaches to postsecondary education. This type 
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of paper often explores rhe meaning and defini- 
tion of competency in the realm of education. 
The category is distinct from descriptive design 
in that it stops short of discussing technical design 
questions or the actual implementation of a par- 
ticular program. 

• History. Papers in this category discuss the ori- 
gins of CBE and how the model has evolved. 

Unlike the research-methodology categories, con- 
tent categories are not mutually exclusive. For instance, 
the study “Assessment Criteria for Competency-Based 
Education: A Study in Nursing Education” eval- 
uates “the effects of the type of assessment criteria 
(performance-based vs. competency-based), the rele- 
vance of assessment criteria (relevant criteria vs. all cri- 
teria), and their interaction on vocational education 
students’ performance and assessment skills.” 19 We 
coded that study’s content under the Assessment and 
Student Outcomes categories because it describes two 
particular approaches to assessment in nursing educa- 
tion and then details how students perform on those 
assessments. 

For an article to be included in a given category, 
the topic had to be a significant focal point of the arti- 
cle. To judge significance, we relied on a list of cues: 
repeated discussion of the topic throughout the study, a 
clear discussion of the topic in the abstract or introduc- 
tory section, or a section devoted entirely to the topic. 
Coding of this sort always entails some level of sub- 
jectivity. To ensure basic consistency at the outset, we 
each coded the same subsample of studies, compared 
our codes, and then refined the coding rules to reflect 
any disagreement. 

Findings 

The first section discusses the distribution of method- 
ologies in the sample of studies. The next moves on to 
examine study content. 

Study Methodology. Of the 380 articles in our analy- 
sis, just over one-quarter (102 studies, or 26.8 percent) 


employed a quantitative design that analyzed data and 
reported statistical results. Sixty percent (228 studies) 
were qualitative, and 1 1.6 percent (44 studies) were lit- 
erature reviews. We were unable to categorize six articles 
(1.6 percent) , as we could not access the full text of each, 
and the abstracts did not clearly reveal the methodology. 

As seen in Figure 1, more than 70 percent of the 
research focused on qualitative research or litera- 
ture reviews. To be sure, qualitative research certainly 
helps guide institutional practice, especially as more 
institutions seek to develop these models. Interview- 
ing students and faculty can provide a more detailed 
and nuanced understanding of how the model works 
and how various stakeholders experience it. It can also 
inform our understanding of how to actually imple- 
ment these types of programs and the potential chal- 
lenges therein. 

For instance, a qualitative study by Jackie Krause, 
Faura Portolese Dias, and Chris Schedler from Cen- 
tral Washington University (CWU) asked how best to 
measure the quality of CBE courses. After reviewing 
the literature on quality assessment in both traditional 
online education and competency-based education and 
gleaning best practices from other CBE providers, the 
authors propose a detailed rubric for quality- assurance 
reviews of online competency-based courses. The 
authors also describe their experience implementing a 
preliminary version of the rubric in the FlexIT program 
at CWU, finding that “many of the existing learning 
objectives needed to be rewritten for competency-based 
courses to be specific and measurable.” 20 The authors 
plan to gather “quantitative data ... for a future research 
study” on the rubric’s broad applicability in other pro- 
grams at CWU and other institutions. 21 

Such studies can inform the development of CBE 
programs and policy reform. In today’s education policy- 
making environment, however, quantitative research is 
the coin of the realm. For instance, in assessing the effect 
of interventions on educational outcomes, the federal 
What Works Clearinghouse considers only quantitative 
studies because, while qualitative designs are useful in 
many respects, they do not allow researchers to assess 
effectiveness from a “credible comparison of outcomes 
between program and counterfactual conditions.” 22 
Our ability to learn more about the model’s effectiveness 
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Figure 1. Distribution of CBE Articles by Article Methodology 


1 . 6 % 



Note: n = 380. 

Source: Authors' calculations. 
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with students from different backgrounds — a key ques- 
tion for policymakers — will depend on quantitative 
research that meets these criteria. 

Most of the quantitative studies were descriptive in 
nature; that is, they use administrative or survey data to 
report observational statistics on student characteristics, 
student perceptions and outcomes, stakeholder opin- 
ions, and other things. For example, Mart van Dinther 
and others survey students’ perceptions of CBE assess- 
ments and measures of student self-efficacy — a belief 
in one’s own effectiveness in a particular area — in their 
article “Student Perceptions of Assessment and Stu- 
dent Self-Efficacy in Competency-Based Education.” 
They examine the interplay of the two indicators, how 
they influence learning outcomes, and which assess- 
ment characteristics in particular influence learning 
positively. 23 

Some quantitative studies focused instead on mea- 
suring CBE’s effect on student outcomes by assessing 
gains in competence made during a course or program. 
Fifteen quantitative studies examined pre- and posttest 
results among a single group of CBE students to assess 
gains in competency. 


For instance, Carrie Rishel and Virginia Majew- 
ski used a pretest-posttest design to measure changes 
in the perceived competency level of 117 students on 
17 competencies at the center of a master’s program 
in social work. The authors also examined self-efficacy 
and found statistically significant effects on 1 7 different 
objectives. The authors argue that their assessment is 
applicable to other CBE programs. 24 Ffowever, with- 
out comparable data from an appropriate comparison 
group, assessing how these gains compare to the rele- 
vant counterfactual of traditional coursework — or no 
coursework at all — is not possible. 

On that score, we found that few studies compared 
the characteristics and outcomes of CBE students to a 
relevant comparison group. Just 13 of the 102 quan- 
titative studies compared the experience and outcomes 
of students enrolled in CBE to a comparison group of 
similar students enrolled in traditional courses. One 
study, “Faculty Collaboration and Competency-Based 
Curriculum Agreements: Meaningful Link in Transfer 
Education,” used a quasi-experimental method to eval- 
uate a competency-based program designed for transfer 
students at five community colleges and one liberal arts 
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Table 1. Percentage of Articles by Content Category 


Number 

Percentage of 

Category 

of Articles 

Total Articles 

Student Characteristics 

13 

3.4% 

Student Outcomes 

60 

15.8% 

Assessment 

56 

14.7% 

Descriptive Design 

178 

46.8% 

Prescriptive Design 

139 

36.6% 

Policy and Market Environment 

101 

26.6% 

Educational Theory 

62 

16.3% 

History 

33 

8.7% 


Note: n = 380. The percentages reflect the proportion of articles falling into each category out of the total number of articles. Percentages do 
not sum to 1 00 percent because an article could be coded as covering more than one category. 

Source: Authors' calculations. 


college. The author found that transfer students who 
came through the CBE route had “higher post-transfer 
GPAs than non-participating transfer students.” 25 An 
older study by Malcolm Taylor compared the program 
outcomes of business school students who earned cred- 
its based on work experience with those who took tradi- 
tional coursework and found that the competency-based 
students outperformed their peers. 26 

Notably, severalstudiesalsoreliedonself- assessments 
of competency — essentially asking students to report 
their level of competence on various dimensions 
before and after the course or program. Such self- 
assessments may reliably estimate actual competency 
levels. But it is not possible to know how reliable they 
are without validating them against an objective mea- 
sure of competence. Even then, competency-based 
assessments would need to be validated against exter- 
nal measures — success in subsequent coursework, 
job placement, employer satisfaction, and so on — 
to ensure that they capture the competencies they 
claim to. 

In short, most of the 380 studies analyzed were 
qualitative in their approach. Quantitative studies were 
less common, and only a handful of quantitative stud- 
ies used data to compare CBE offerings to traditional 
courses or programs. Therefore, we still lack answers to 
basic questions that could help guide policy. 


Study Content. When it comes to the content of the 
articles in our sample, the most common categories 
focused on program design — either describing what 
programs currently do (descriptive) or how practi- 
tioners should design their programs (prescriptive). 
Recall that studies could fall into more than one cate- 
gory, so the percentages do not sum to 100. 

Nearly 47 percent of the studies had a descrip- 
tive design component, and about 37 percent pro- 
vided a prescriptive discussion of program design. A 
smaller — although not insignificant — number of stud- 
ies included analyses of student outcomes or assessment 
techniques (15.8 and 14.7 percent, respectively). About 
one-quarter of the studies featured a discussion of the 
policy and market environment for CBE. Thirteen arti- 
cles analyzed the characteristics of students enrolled in 
CBE programs. 

To provide more detail, we also looked at the 
cross-tabs of methodology and content. What kind 
of qualitative and quantitative research has been con- 
ducted on what topics? Table 2 provides the cross-tabs 
of methodology by selected content categories. 
(Table A1 in the appendix includes the full set of 
cross-tabs.) 

Table 2 shows that of the 228 qualitative articles, 
about half described the design of current CBE pro- 
grams (113 articles), while 97 articles prescribed design 
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Table 2. Cross-Tab Analysis of Article Content by Article Methodology 



Total 

Articles 

Student 

Characteristics 

Student 

Outcomes 

Assessment 

Descriptive 

Design 

Prescriptive 

Design 

Policy and 
Market 
Environment 

Quantitative 

102 

8 

42 

35 

47 

35 

15 

Qualitative 

228 

4 

16 

20 

113 

97 

69 

Literature Review 

44 

0 

2 

1 

15 

5 

14 


Notes: n = 374. Uncategorized articles (by methodology) are not included in this table. Full table with all content categories available in Table 
A1 in the appendix. 

Source: Authors' calculations. 


principles — “best practices” — for designing or improv- 
ing a CBE program. In a 2014 Change Magazine article, 
for instance, Kris Clerkin and Yvonne Simon outlined 
the features of Southern New Hampshire University’s 
competency-based program, from its project-based cur- 
riculum with learning coaches to its close cooperation 
with employers and workforce-relevant competencies. 27 

Another Change Magazine article from Sally M. 
Johnstone and Louis Soares provides a prescriptive take 
on how to design a CBE program. The paper argues 
that CBE “can fit into existing campus structures, 
if certain principles are followed: the degree reflects 
robust and valid competencies; students are able to 
learn at a variable pace and are supported in their learn- 
ing; effective learning resources are available any time 
and are reusable; [and] assessments are secure and reli- 
able.” 28 The authors say that the principles “can help 
guide higher education leaders as they develop their 
own CBE programs.” 20 

As you might imagine, these two categories often 
overlapped. Indeed, 27 articles included both a 
descriptive and prescriptive discussion of CBE pro- 
gram design. One such study, which examines the 
University of Missouri’s competency-based dental 
hygiene program, provides “examples of program 
competencies . . . the complete scoring rubric for 
portfolios, and the student portfolio evaluation sur- 
vey.” 30 The article concludes that “although port- 
folio evaluation requires continual revision and 
analysis, it offers many advantages” and suggests 
how providers can adopt the university’s approach to 
competency-based assessment. 31 


Turning now to the 102 quantitative studies, 
Table 2 shows that many of these articles examined stu- 
dent outcomes in some respect. Forty-two articles were 
outcomes-focused, a much higher proportion than in 
the qualitative studies. 

Many of these articles analyzed the assessment scores 
of students in a particular competency-based program 
or discipline. For instance, in their paper “An Integrated 
Competency-Based Approach to Management Educa- 
tion: An Italian MBA Case Study,” Arnaldo Camuffo 
and Fabrizio Gerli measure student gains in 27 sepa- 
rate competencies over the course of their MBA at an 
Italian school of management. 32 The authors exam- 
ine self-assessments and behavioral event interviews 
from before and after the program and found signifi- 
cant gains in many areas. However, the study did not 
include a comparison group of non-CBE students. 

Only a small number of studies focused on retention 
or graduation rates in CBE programs; two are worth 
highlighting. One study by Miriam Ben-Yoseph, Patrick 
Ryan, and Ellen Benjamin measured the retention and 
graduation rates of adult students in a competency-based 
individualized degree program. Using enrollment data 
from 1986 to 1996 and course-progression data from 
1990 to 1995, the authors found that “three-fourths 
of those who graduated 1988-94 did so within three 
years; retention was most difficult in the self-directed 
and research phases [of the program]; [and] the wom- 
en’s graduation rate was higher.” 33 

Another study from Milan Hayward and Mitch- 
ell Williams examined differences in graduation rates 
at four community colleges among adult learners who 
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used prior-learning assessments (PLA), those who did 
not, and those who used different methods of PLA. The 
researchers found significant differences in completion 
rates for PLA learners (28.4 percent) and non-PLA 
learners (11.8 percent), as well as significant differences 
among various distinct PLA methods. 34 Unfortunately, 
studies employing rigorous designs such as these were 
few and far between. 

Only a few quantitative studies examined labor mar- 
ket success, despite high hopes that CBE will send a 
clearer signal to the labor market. Just 14 articles in the 
Student Outcomes category mentioned employers or 
industry. Some of these articles focused on employers’ 
role in identifying competencies and informing pro- 
gram design. Rarely, though, did researchers actually 
follow graduates into the labor market, looking at job 
placement or career readiness. 35 

Just one study looked at employment rates. It exam- 
ined the relationship between a competency-based lan- 
guage program for immigrants in Australia and job 
attainment, finding a significant but weak relationship 
between the students’ level of English-language compe- 
tence and employment. 36 

Several studies used surveys of subject-matter 
experts or employers to identify the key competencies 
for a given program of study. A common approach 
in these studies was the so-called “Delphi method,” 
in which experts are surveyed numerous times and 
are provided with a summary of interim, aggregate 
results from the first questionnaire before answering 
the second. The idea is to encourage experts to revise 
their initial answers based on information from their 
peers, thereby illustrating areas of consensus and dis- 
agreement. 37 We coded these studies as falling into 
the “quantitative” and “prescriptive design” cells; they 
used quantitative survey data to identify key compe- 
tencies and typically made clear curricular recommen- 
dations based on their findings. 38 

A similar number of quantitative studies (35 stud- 
ies) focused on the assessments at the heart of CBE 
models — a research area that is crucial to building con- 
fidence in the competency-based approach. Many of 
these studies focused on validating competency mea- 
sures to ensure that they accurately captured what they 
intended. 


In one example, Marion Bogo and coauthors eval- 
uated the predictive validity of a new objective struc- 
tured clinical evaluation (OSCE) tool, an assessment 
for a practicum in a master’s in social work program. 
The researchers administered a one-scenario OSCE to 
125 students, rating their performance against previ- 
ously standardized scales for the practicum. They found 
that the OSCE captured a wide range of scores associ- 
ated with performance in the practicum, but also that 
some students who performed poorly on the OSCE 
performed well in the practicum evaluation. 39 

One area that received consistent attention from 
researchers was the Policy and Market Environment 
category. Roughly a quarter of all articles discussed this 
topic, with 69 qualitative and 15 quantitative articles 
covering this domain. Studies in this category looked 
at the different types of CBE emerging across higher 
education (direct assessment, PLA) and key policy 
questions, such as CBE’s relation to federal credit-hour 
policies and accreditation. 40 

Notably, 62 of the 84 studies in this category exam- 
ined CBE in a foreign context. Overall, few studies 
in this category systematically examined the domestic 
landscape of CBE providers. As a result, it is difficult 
to get a comprehensive list of who is offering CBE, let 
alone who enrolls, how much they pay, and how well 
they do. 

Foreign studies were common across the board; 
48.2 percent of all articles studied (183) examined 
CBE in another country. The most frequent foreign 
countries studied were Australia (37 articles), England/ 
United Kingdom (29 articles), Canada (25 articles), 
and the Netherlands (21 articles). In all, 29 distinct 
countries were mentioned, not counting the United 
States. 41 While much of this research may be applica- 
ble to the US context, studies of American programs 
are likely the most relevant to current policy debates. 

What Questions Should Researchers 
Ask About CBE? 

In sum, we still know less than we need to about 
how many institutions are offering CBE, who they 
serve, and how they perform. When it comes to 
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questions of student outcomes, the picture is seri- 
ously incomplete. 

Some of this is obviously due to the recent spike in 
the creation of competency-based programs. Early on, 
practitioners and affiliated researchers are largely focused 
on trying to develop and implement effective programs, 
and descriptive work can help others identify the best 
practices. Likewise, formal evaluations of the model take 
time. Completion rates and labor market success take 
years to measure, and it is likely best to wait until a pro- 
gram is fully implemented before evaluating it. All of this 
means that many new programs could be a few years 
away from a comprehensive study of their outcomes. 

At the same time, lawmakers and regulators will 
naturally ask whether the model “works” and for 
which types of students. These policymakers will be 
hard-pressed to find the kind of rigorous evaluation 
that has become increasingly common in other areas of 
education policy. 

But researchers, funders, and policymakers have 
an opportunity to build such a research base. What 
research questions should they pursue? And what roles 
can different stakeholders play in promoting such a 
research agenda? 

Areas for Future Research. A new research agenda 
should augment existing descriptive, programmatic 
work with empirical studies of the following questions. 

How do CBE students compare to those enrolled in tra- 
ditional programs ? Conventional wisdom suggests that 
CBE programs are well-suited for and tend to attract 
nontraditional students — working adults who often 
have the opportunity to gain credit for prior learn- 
ing. But is this conventional wisdom accurate? Which 
types of students select into CBE programs, and how 
do they compare to the students who enroll in tradi- 
tional courses and programs? Basic descriptive studies 
of student characteristics, perhaps coupled with sur- 
vey data on incoming student perceptions and goals, 
would do much to inform our understanding of the 
CBE landscape. 

How do measures of student success — learning progress, 
completion, and employment — in CBE programs compare 


to those in comparable traditional programs ? The major 
blind spot in CBE research relates to student outcomes. 
Unlike other innovations — online learning, for one — 
CBE has not yet been subject to the sorts of rigorous 
evaluations in which outcomes of “treated” students 
(those enrolled in CBE) are compared to outcomes of 
students in a “control” condition (traditional course- 
work, perhaps). 

The gold standard here is a randomized study, 
although observational studies that rely on carefully 
considered comparison groups could also inform the 
discussion. This research should provide more than 
a simple “horse race” between CBE and traditional 
coursework. Rather, it should focus on asking how dif- 
ferent types of students fare when they enroll in differ- 
ent varieties of CBE, as compared to similar students 
who either enroll in traditional coursework or do not 
enroll at all. Learning which models seem to be more 
effective and with which types of students can help pol- 
icymakers and school leaders target their efforts where 
they are most likely to be successful. 

Do the assessments at the heart of CBE map to success in 
future tasks (subsequent courses, graduate school, or the 
workplace )? Another related question is how well per- 
formance on CBE assessments predicts future suc- 
cess — are the assessments “externally valid”? As Katie 
McClarty and Matthew Gaertner asserted in a recent 
paper, “External-validity evidence is critical to support- 
ing the claims that CBE programs can make about the 
relationship between their measures of competence and 
workplace success, and about comparability of gradu- 
ates from CBE and non-CBE programs.” 42 Research- 
ers should design studies that test the predictive power 
of CBE assessment scores with students’ performance 
in higher-level courses in the same program, graduate 
school courses, or the labor market. 

Is CBE more cost-effective than traditional models for 
students and institutions )? One of the most common 
arguments in favor of CBE focuses on its apparent 
affordability. The model itself is likely less expensive 
to offer than traditional coursework; rather than sit- 
ting through a set number of lectures and sections, stu- 
dents can learn at their own pace and take a series of 
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assessments. The ability to move more quickly — espe- 
cially under “all-you-can-learn” tuition plans — may 
lower time to degree and tuition paid. 

As Robert Kelchen points out in his analysis of tui- 
tion pricing in CBE, while earning more affordable 
credits through CBE programs is certainly possible, it 
is not a foregone conclusion. 45 This is a simple enough 
empirical question: how much do CBE graduates pay, 
on average, for their degrees relative to comparable stu- 
dents who earn degrees in traditional programs? How 
much debt do these two groups of students take on? 
This information is readily available via administra- 
tive data, and a descriptive research project would help 
inform the conversation. 

How do employers view CBE ? In “Employer Perspec- 
tives on Competency-Based Education,” an earlier AEI 
paper in this series, Parthenon-EY’s Chip Franklin and 
Rob Lytle surveyed a sample of employers and gener- 
ally found an openness to CBE models but also low 
levels of overall awareness of the approach. 44 But there 
is much more work to be done. 

A more detailed look could identify employers that 
have hired a critical mass of CBE graduates and ask 
them how they view those employees compared to 
those who came from more traditional routes. Research- 
ers could couple survey data with objective measures 
(earnings, advancement). In addition, researchers could 
conduct a “resume experiment,” in which they send fic- 
titious resumes to employers with job openings, some 
of which feature a CBE degree or transcript (the treat- 
ment group), while the others feature a traditional 
degree or transcript. 45 Does the additional information 
that comes with a CBE credential affect employer deci- 
sions about who to interview and hire? These research 
questions could help fill in some of the blanks in the 
study of student outcomes. 

How Can Stakeholders Support This Research 
Agenda? Who can help push this research agenda 
along? Thanks to the quick and continued expansion 
of CBE, there is a tremendous opportunity to build 
a sustainable research and development enterprise 
that can inform institutional practice and federal and 
state policy. 


The federal government could play a significant role 
in expanding the study of this new mode of instruc- 
tion, particularly through the Department of Educa- 
tions ongoing Experimental Sites Initiative. Under 
this program, the department waives certain eligibil- 
ity requirements for the federal student aid program, 
allowing a set number of new, innovative providers to 
receive federal grants and loans. CBE is now the focus 
of an Experimental Sites project, and 40-plus programs 
were accepted into the program last year. 46 

Using experimental-sites authority to test CBE 
models is a positive step. However, policymakers must 
make sure that this flexibility is coupled with rigorous 
evaluation, ideally modeled on the What Works Clear- 
inghouse standards. 47 Congress could also consider set- 
ting up a CBE demonstration project, as Reps. Jared 
Polis (D-CO) and Matt Salmon (R-AZ) proposed in 
their 2014 bill. 48 The demonstration project bill would 
change federal law, officially allowing the secretary of 
education to give waivers to up to 30 CBE providers 
for participation in the federal student aid program. 
Demonstration projects should be subject to rigorous 
evaluations as well. 

Philanthropy also has an obvious role to play here. 
Major higher education funders have shown interest 
in the model and have invested in efforts to build 
programs, study the policy environment, and so 
on. 49 These gifts have likely played an important role 
in encouraging the recent bump in interest in CBE. 
Philanthropists could build on this work by fund- 
ing rigorous quantitative work on the questions laid 
out earlier. 

In September 2015, the tandem of the Bill & 
Melinda Gates Foundation and the Lumina Founda- 
tion released a report on CBE providers’ roles in the new 
experimental-sites initiative, one of which included vol- 
unteering for “vigorous, third party evaluation . . . and 
[leading] a national research agenda ... to ensure that 
the effectiveness of CBE programs will be rigorously 
evaluated.” 50 These organizations should continue to 
fund projects that fulfill this spirit. 

Finally, higher education researchers should take a 
fresh look at these important research questions, espe- 
cially researchers who are not immediately part of the 
CBE community. As it happens, WGU and publisher 
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John Wiley & Sons recently founded the Journal of 
Competency-Based Education, a new peer-reviewed jour- 
nal focused explicitly on CBE. (Full disclosure, one of 
us sits on the advisory board for the journal.) The goal 
is to advance theoretical and empirical research on the 
model, and its first issue featured original research on 
student and faculty experiences, student outcomes, and 
costs. The journal will also feature case studies on exist- 
ing programs’ practices and short opinion essays on the 
state of CBE . 51 This journal could provide academics 
with a peer-reviewed platform to publish new research 


while creating a clearinghouse for work that tackles 
many of the earlier questions. 

Attention to the research questions and the vari- 
ous stakeholder roles laid out earlier will help build 
our understanding of CBE programs, but only if such 
research is truly objective and open-ended. Wherever 
possible, CBE proponents and sympathizers should 
invite outsiders in to apply their tools to the model. 
Doing so will enhance the credibility of the research 
and increase the likelihood that policymakers will come 
to rely on it. 


Appendix 


Table A1. Full Cross-Tab Analysis of Article Content by Article Methodology 



Total 

Articles 

Student 

Charac- 

teristics 

Student 

Out- 

comes 

Assess- 

ment 

Descrip- 

tive 

Design 

Prescrip- 

tive 

Design 

Policy and 
Market 
Environment 

Edu- 

cational 

Theory 

History 

Quantitative 

102 

8 

42 

35 

47 

35 

15 

7 

0 

Qualitative 

228 

4 

16 

20 

113 

97 

69 

33 

15 

Literature Review 

44 

0 

2 

1 

15 

5 

14 

21 

18 


Notes: n = 380. Uncategorized articles (by methodology) are not included in this table. 
Source: Authors' calculations. 
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